Module 9: Observability and Monitoring

Monitoring Evolves into Observability

Distributed serverless applications make observability even more critical - no servers to SSH into, many small components.

Three Pillars of Observability

Amazon CloudWatch Logs

• Centralizes logs from Lambda, API Gateway, and all AWS services
• Log Groups = container (e.g., /aws/lambda/my-function)
• Log Streams = sequence of events from one source
• Logs Insights = interactive query language (SQL-like)
• Metric Filters = generate metrics from log patterns
• Retention policy = set expiry (default: forever!)

API Gateway Logging

• Execution logs (REST only): ERROR or INFO level, full request/response
• Access logs (REST + HTTP): every request, customizable format

Structured Logging Best Practices

# Python - structured JSON logging (recommended!)
import json

message = {
    "level": "INFO",
    "timestamp": "2024-12-11T12:44:40.300Z",
    "requestId": "abc-123",
    "orderId": "ORD-456",
    "action": "AddToCart",
    "quantity": 2,
    "productId": "a23390f3",
    "environment": "prod"
}
print(json.dumps(message))

# CloudWatch Logs Insights can then query:
# fields @timestamp, orderId, action
# | filter level = "ERROR"
# | sort @timestamp desc
# | limit 20

AWS X-Ray - Distributed Tracing

• One-click enablement for Lambda + API Gateway
• Visual service map shows request flow + latency
• Trace = end-to-end request journey across services
• Segments, subsegments, annotations, metadata

Instrumenting with X-Ray SDK

# Python - instrument your Lambda with X-Ray
from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all

# Patch all AWS SDK clients to capture traces automatically
patch_all()

def handler(event, context):
    # All boto3 calls are now traced automatically!
    # Add custom subsegments for your own code:
    subsegment = xray_recorder.begin_subsegment('process_order')
    try:
        result = process_order(event)
        subsegment.put_annotation('orderId', event['orderId'])
        subsegment.put_metadata('result', result)
    finally:
        xray_recorder.end_subsegment()
    return result

Amazon CloudWatch Metrics

• Default metrics - AWS services emit automatically (Invocations, Duration, Errors, Throttles)
• Custom metrics - your business data (orders/min, conversion rate, revenue)
• Alarms - trigger actions when metric crosses threshold (SNS, Auto Scaling, Lambda)
• Dashboards - visualize metrics in real-time

Key Lambda Metrics to Monitor

Custom Metrics & Embedded Metrics Format

# Embedded Metrics Format (EMF) - generate metrics from logs!
# No putMetricData API call needed - just print structured JSON

import json
from datetime import datetime

metric_log = {
    "_aws": {
        "Timestamp": int(datetime.now().timestamp() * 1000),
        "CloudWatchMetrics": [{
            "Namespace": "MyApp/Orders",
            "Dimensions": [["Environment", "Region"]],
            "Metrics": [
                {"Name": "OrderCount", "Unit": "Count"},
                {"Name": "OrderValue", "Unit": "None"}
            ]
        }]
    },
    "Environment": "prod",
    "Region": "us-west-2",
    "OrderCount": 1,
    "OrderValue": 149.99
}
print(json.dumps(metric_log))
# CloudWatch automatically extracts OrderCount and OrderValue as metrics!

CloudWatch Lambda Insights

• Enhanced monitoring via a Lambda Layer (opt-in per function)
• Collects system-level metrics: CPU, memory, network, disk
• Identifies cold starts, memory leaks, timeout patterns
• Performance dashboard per function out of the box

Key Insight Metrics

Bringing the Three Pillars Together

The X-Ray trace map integrates X-Ray and CloudWatch, providing access to logs, metrics, and alarms from a single interface.

What's New (2024-2025)

• Advanced Logging Controls - JSON format, log-level filtering (INFO/WARN/ERROR) per function
• CloudWatch Application Signals - Auto-discover services and SLOs without instrumentation
• X-Ray + OpenTelemetry - ADOT collector for vendor-neutral instrumentation
• CloudWatch Anomaly Detection - ML-based alerting without manual thresholds
• Lambda Powertools Tracer - Decorator-based X-Ray tracing with zero boilerplate
• Logs Insights Improvements - New query functions, correlation with traces

Live Demo: Observability in Action

Deploy a Lambda + API Gateway with full observability, then show logs, traces, and metrics.

Demo Step 1: Deploy Function with X-Ray + Logging

Lambda Function (with structured logging + X-Ray)

# observability_demo.py
import json, os, time, boto3
from aws_xray_sdk.core import xray_recorder, patch_all

patch_all()  # Auto-trace all AWS SDK calls

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table(os.environ.get('TABLE_NAME', 'demo-orders'))

def handler(event, context):
    start = time.time()
    order_id = event.get('queryStringParameters', {}).get('id', 'ORD-001')

    # Structured log (JSON)
    print(json.dumps({"level": "INFO", "action": "GetOrder", "orderId": order_id}))

    # Custom X-Ray subsegment
    with xray_recorder.in_subsegment('fetch_order') as subseg:
        subseg.put_annotation('orderId', order_id)
        item = table.get_item(Key={'orderId': order_id})

    duration = (time.time() - start) * 1000
    print(json.dumps({"level": "INFO", "action": "Complete", "duration_ms": round(duration)}))

    return {"statusCode": 200, "body": json.dumps(item.get('Item', {}))}

Demo Step 2: Deploy with CLI

# Create function with X-Ray enabled
zip observability_demo.zip observability_demo.py

aws lambda create-function --function-name observability-demo \
  --runtime python3.12 --handler observability_demo.handler \
  --role arn:aws:iam::ACCOUNT:role/lambda-xray-role \
  --zip-file fileb://observability_demo.zip \
  --environment Variables='{TABLE_NAME=demo-orders}' \
  --tracing-config Mode=Active \
  --logging-config LogFormat=JSON,ApplicationLogLevel=INFO

# Create DynamoDB table
aws dynamodb create-table --table-name demo-orders \
  --attribute-definitions AttributeName=orderId,AttributeType=S \
  --key-schema AttributeName=orderId,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST

# Put test item
aws dynamodb put-item --table-name demo-orders \
  --item '{"orderId":{"S":"ORD-001"},"item":{"S":"Laptop"},"amount":{"N":"999"}}'

# Create HTTP API + invoke
aws apigatewayv2 create-api --name observability-demo-api --protocol-type HTTP
# (attach integration + route, then test)

# Invoke directly to generate traces
aws lambda invoke --function-name observability-demo \
  --payload '{"queryStringParameters":{"id":"ORD-001"}}' output.json

Demo Step 3: What to Show in Console

Logs Insights Query to Run Live

fields @timestamp, action, orderId, duration_ms
| filter level = "INFO"
| sort @timestamp desc
| limit 20

Module Summary

• Distributed serverless increases the importance of observability
• Three pillars: Logs (what), Traces (where), Metrics (how)
• CloudWatch Logs: centralized, queryable with Logs Insights
• X-Ray: visual trace maps, latency breakdown, annotations
• CloudWatch Metrics: alarms, dashboards, Lambda Insights
• EMF: generate metrics from structured logs automatically
• Set log retention policies (default = forever = expensive!)
• Use Powertools for structured logging + tracing with decorators